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TESTING IF TWO MEASURING PROCEDURES MEASURE 



THE SAME PSYCHOLOGICAL DIMENSION 
Abstract 

A simple, rigorous, small-sample statistical technique is described 
for testing the hypothesis that two sets of measurements differ only 
because of errors of measurement and because of differing origins and 



units of measurement. 
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TESTING IF TWO MEASURING PROCEDURES MEASURE 
THE SAME PSYCHOLOGICAL DIMENSION* 

This note is concerned with testing the hypothesis that two sets of 
measurements differ only because of a) errors of measurement, b) differing 
units of measurement, and c) differing arbitrary origins for measurement. 

It describes a simple, rigorous, small-sample statistical technique that, 
unnoticed, has become available (Villegas, 1964). Other techniques used 
for this purpose are cumbersome, approximate, or flawed (Forsyth & Feldt, 
1969, 197O ; McNemar, 1958; Lord, 1957)-. 

Statistical Procedure 

The necessary raw data consist of r > 2 replicate measurements (in- 
dexed by the subscript k ) on each of N people (indexed by a ) by each of 
two measuring instruments or procedures (indexed by g or h ). For each 
instrument or procedure separately, compute the usual among -persons and 
within-persons sums of squares. Also the corresponding sums of cross 
products b'etween instruments (or procedures). Let W denote the 2-by-2 
matrix of within-persons sums of squares and cross products, and let A 

denote the corresponding 2-by-2 among-persons matrix. In a standard 

N r , 

notation, the element of W in column g and row h is E ,1 (x 

Q, — JL K— _L 

- x^ a )( x haV - x^.) ’ A typi 08 - 1 element of A , similarly, is 

rZ Ll (x ga- ' X g-- )(x ha- - Vl ' 

*This research was sponsored in part by the Personnel and Training 
Research Programs, Psychological Sciences Division, Office of Naval 
Research, under Contract No. N00014-69-C-0017, Contract Authority 
Identification Number, NR No. 150-303, and Educational Testing Service. 
Reproduction in whole or in part is permitted for any purpose of the 
United States Government. 
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Let Fp denote the 1 - p percentile of the F distribution with 
N and N(r - l) degrees of freedom, for the numerator and denominator 
respectively. Compute the matrix 

M = (r - l)A - F W 
P 

From the product of the two diagonal terms of M subtract the 
product of the two off-diagonal terms, thus obtaining the value of the 
determinant j M | If the determinant is positive and if both diagonal 
terms are also positive, then the F test rejects at significance level 
p the hypothesis EL stated at the beginning of this paper. Otherwise, 
the hypothesis is not rejected. This same conclusion may be stated more 
compactly as follows: EL is rejected if and only if the matrix M is 

positive definite. 



Discussion 

The assumptions required for the validity of the foregoing statistical 
procedure and conclusions are as follows. Within each replication, the 
errors of measurement for instruments g and h are random variables 
having a bivariate normal distribution. The mean error is always taken to 
be zero. No restrictive assumptions are made about the variance or cor- 
relation parameters of this distribution; in particular, the errors for g 
and h may be correlated within a replication. The within-replication 
bivariate distribution is the same for each replication and for each 
individual measured. Across replications, the errors of measurement are 
always uncorrelated with each other. 
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The unknown relationship between the two origins mentioned in , 
also that between the two units of measurement, represent "nuisance param- 
eters" for the purpose of making a significance test of H Q . Villegas 
shows that when (and only when) M is positive definite, there is no set 
of values for these (unknown) nuisance parameters such that an appropriate 
variance ratio calculated from them and from the data will lie below the 
p -level of significance. 

Further developments along similar lines are reported by Walter 
Kristof (personal communication). 

It is important in practice that the replications meet the assumptions 
stated. Any practice effect between replications that increases the within- 
persons sums of squares, for example, will tend to decrease the chance of 
rejecting . 

It is worth noting that Tukey (1951 ) accurately foretold the nature 
of Villegas' significance test. Tukey describes a very similar test and 
states 

"While not identical, the correction of correlation coefficients 
for attenuation is extremely close to our topic .... (The writer 
believes that correlation coefficients have been highly 'efficient' 
in shielding the essentials of this and related situations from 
the inquiring eye. Unless and until we come to variances and 
covariances in measured, not standardized units, it is hard to 
meet the facts face to face.)" 



Numerical Example 

In the numerical example, measuring instruments g are unspeeded 
15 -item vocabulary tests; measuring instruments h are highly speeded 
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75-item vocabulary tests. The r = 2 parallel forms of each test were 
administered to each of N = 649 examinees. The raw data are the same 
as tho&e used in a previous numerical example (Lord, 1957, PP- 210-212). 
The required s ums of squares and products are shown in the accompanying 
table . 



Among Persons 
Within Persons 
Total 



Unspeeded Test 

Sum of 
Squares 

93524.56 



Sum of 
Products 

76176.30 



Speeded Test 

Sum of 
Squares 

111220.58 



18533.00 



- 547.00 15405.50 



112057-56 



75629.30 



126624.08 



The .05 significance level of F for 649 and 649 degrees of freedom 
is 1.13* The matrix M is easily found to be 



M = 



"72582 

.76794 



76794 ' 

93815, 



The determinant is positive, so EL is rejected. This agrees with the 
conclusion reached previously by large-sample methods (Lord, 1957) under 
somewhat different assumptions. 
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